IMPROVED METHOD FOR DIAGNOSING COMPLEX SYSTEM FAULTS 



BACKGROUND OF THE INVENTION 

5 

Over recent years, several methods for diagnosing faults in systems comprised 
of multiple components have been devised. Typically, such methods utilize 
information concerning the structure of the system, the types of tests executed on the 
system, and the results of those tests in order to identify the system component most 

10 likely responsible for any test failures encountered. 

Such diagnostic methods often have been applied to electronic systems at 
varying levels of abstraction. For example, a defective computer among hundreds or 
thousands of computer systems constituting a computer network may be identified by 
such a method. Similarly, the same method may be employed to identify the source 

15 of faults within a single computer, printed circuit board, or integrated circuit. 

Furthermore, such methods may be utilized in systems residing outside the electronics 
domain, including, but not limited to, mechanical, aeronautical, and medical or 
biological systems. 

A diagnostic system employing such a method has been described in U.S. 

2 o Patent Number 5,808,9 1 9 by Christopher Preist and David Allport, which is assigned 
to the assignee of the present invention. Generally speaking, such a system begins its 
diagnosis by generating sets of logically possible diagnoses. The system 
accomplishes this task by first identifying "conflict sets," wherein each set identifies 
all of the components involved in a particular set of failing functional tests. 

2 5 Conceptually, each test, whether failing or passing, employs one or more 

"operations," or functions, with each operation employing one or more physical 
"components" of the system. Each component is exercised by an operation at a 
particular level of "utilization," or coverage, represented as a fraction between 0 and 1 
inclusive, with 0 indicating no utilization, and 1 representing complete utilization of 

3 0 the component. Using these conflict sets, the system then generates a number of 
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"hitting sets," with each representing a set of components that, if defective, would 
result in the identified set of failing functional tests. Therefore, each hitting set 
represents a logically possible diagnosis, given the results of failing tests. 

Once each of the possible hitting sets, or candidate diagnoses, is identified, 
5 each is assigned a relative weight by way of calculation of various probabilities of 
failure. The basis of the calculations is Bayes' Rule: 

p(D|T) = (p(D)*p(T|D))/p(T), 

10 Using this rule, D = a candidate diagnosis and T = the complete set of failing 

and passing test results. p(D|T) is the probability of the components of the candidate 
diagnosis failing, given the set of test results. This value is also known as the 
"posterior probability" of the candidate diagnosis. Alternately, p(D) is the "prior 
probability" of the components of the candidate diagnosis failing when no 

15 information concerning test results is considered. Additionally, p(T|D) is the 

probability of a given set of test results, given a particular candidate diagnosis is 
correct. Finally, p(T) is the probability of a particular set of test results regardless of 
the correct candidate diagnosis. 

Typically, the specific value of the posterior probability of the candidate 

2 o diagnosis is not required; what is important is the relative ranking of the posterior - 
probability for each candidate diagnosis so that the diagnosis most likely responsible 
for the observed test results is identified. Therefore, since p(T) is the same for each 
candidate diagnosis, that term may be canceled from p(D|T) for each candidate 
diagnosis, leaving a relative posterior probability: 

25 

Relative p(D|T) = p(D) * p(T|D) 

To calculate the first factor, p(D), an assumption is normally made that each of 
the components of the candidate diagnosis fail independently. As a result, the prior 
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probability of the candidate diagnosis may be calculated by multiplying together the 
failure rate of each component of the candidate diagnosis: 

P(P) = 11 (failure rate (component)) 

^ VcomponentseD 

(In the previous equation, as well as in ones to follow, "V" symbolizes "for 
all," and "e" signifies "element(s) of") 

The failure rate of each component employed is usually provided by the 
10 manufacturer of that particular component, based on empirical evidence over many 
thousands of components produced. Additionally, the failure rate may also include 
other sources of error, such as "process errors" (e.g., the improper placement or 
soldering of the component to a printed circuit board). 

To calculate p(T|D) for a particular candidate diagnosis, the diagnosis system 
15 typically assumes that the results for each test are independent to simplify the 

calculations involved. In that case, the probabilities of each test result for a candidate 
diagnosis are simply multiplied. For example, if two tests, Ti and T2, are involved, 
the resulting p(T|D) would be the product of the probabilities involving each test: 

20 p(T|D) = p(T 1 |D)*p(T 2 |D) 

To further reduce the complexity of the calculations involved, an additional 
assumption is normally made that the probability of a test failing, given the failure of 
a particular component of a candidate diagnosis, is proportional to the utilization of 
25 that component by that particular test. Likewise, the probability of a test passing, 
given the failure of a component, is assumed to be (1 - the utilization of that 
component by the passing test). For example, if Ti is a failing test, T 2 is a passing 
test, and the candidate diagnosis is a single component d, the posterior priority of the 
overall test results, given a candidate diagnosis, may be stated as follows: 
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p(T|D) = utilization of Ci in Tj * (1 - utilization of Ci in T 2 ) 



In some alternate applications, the utilization of the component of a failing test 
is assumed to be one, further simplifying the overall calculation at the possible 
5 expense of some inaccuracy in determining the relative posterior probability for each 
hitting set. Such a simplification is possible if the component involved is simple 
enough that any operation exercising it employs essentially the entirety of the 
component. 

In the case where a candidate diagnosis consists of multiple components, the 
10 utilization of each of the components by each of the failing tests, and one minus the 
utilization of the each component by each passing test, may be factored together to 
generate p(T|D). 

Once p(D) and p(T|D) are determined for a candidate diagnosis (i.e., hitting 
set), they are multiplied together to obtain the relative posterior probability p(D|T) for 

is that diagnosis, as shown above. p(D|T) for each diagnosis is then transformed to a 
final relative weight by the possible application of an "operation penalty." The 
operation penalty is applied if the diagnosis involves operation violations, which are 
operations which are involved in both passing and failing tests, thus implying an 
inconsistency in the test results. In those cases, the weight of that diagnosis is 

20 penalized, or reduced, to reflect that inconsistency. 

To determine if the operation penalty, which is typically a single scalar factor 
less than one sometimes called the "input variability," is to be applied to the weight of 
a particular candidate diagnosis, each failing test T\ is analyzed in turn. Generally 
speaking, the input variability is a measure of the expected variability of results over 

25 multiple uses of the same operation. In other words, the more repeatable the results of 
a particular operation, the lower the input variability. Each operation of Ti that 
involves a component of the candidate diagnosis is identified. If each of those 
operations is also involved in a passing test, the operation penalty is to be applied. 
Once a failing test Tj fits this description, the operation penalty is applied only once; 
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only if none of the failing tests indicate application of the penalty is the penalty not 
applied. 

Once the relative weight of each of the candidate diagnoses is calculated, the 
diagnosis with the highest relative weight is the one determined by the diagnostic 
5 system as being the most likely cause of the failures detected by the tests (i.e., the 
correct diagnosis). 

In some alternative versions, no assumption is made that the probability of a 
test failing, given the failure of a particular component, is proportional to the 
utilization of that component by the test. In that case, the relative weight of a 
10 candidate diagnosis, given a set of test results, may be defined as follows: 

Weight(D,T) = p(D) * minimum (<x\, i = 1 to N), wherein 
cti = 1 - utilization of component Cj by test i, where i is a passing test, and 
Cj is a member of D, or 
is a* = 1 where i is a failing test, and 

N = the total number of passing and failing tests 

Therefore, the maximum utilization of a component in any passing test is used 
to alter the weight of the candidate diagnosis. Also, the actual utilization of a 
2 o component in a failing test is not considered. 

In the case of a multiple-component diagnosis, p(D) is sometimes replaced by 
the product of the prior probabilities of each component Cj. The utilization factor 
may be replaced, for example, by the minimum or average utilization for each 
component Cj. 

25 Additionally, systems under test that include subcomponents of components 

may be diagnosed. Since a component is viewed as the smallest portion of the system 
that may be replaced upon failure, the failure of any subcomponent of a component 
would necessitate replacement of the entire component. In this situation, the weight 
of each component is calculated separately. Afterward, the relative weight of the 
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component as a whole is then the maximum of the weights calculated for each of the 
component's subcomponents separately. Subcomponents residing outside a defined 
hitting set or candidate diagnosis are not considered in determining the weight of that 
diagnosis. 

5 Although diagnostic systems of this type have been extremely useful in many 

cases, some systems exhibit failures that have proven difficult to diagnose. For 
example, some radio- frequency (RF) devices exhibit complex faults that often result 
in seemingly conflicting passing and failing test results, resulting in operation 
penalties being assessed for multiple candidate diagnoses, often causing those 
10 diagnoses to be equally weighted. Such weighting leads to reduced guidance 

concerning which components of the system to replace first. Additionally, some tests 
Utilizing a particular operation may pass or fail intermittently due to the uncertain 
nature of the specific measurement involved with those tests, thereby skewing the 
weighting. 

is From the foregoing, a need exists for a more reliable method of diagnosing 

faults in systems that is able to distinguish between candidate diagnoses when 
complex faults, often resulting in confusing and conflicting test results, are involved. 



2 o SUMMARY OF THE INVENTION 

Embodiments of the invention, to be discussed in detail below, provide an 
improved method for diagnosing faults of a system under test. One or more candidate 
diagnoses, each of which is potentially responsible for failures observed during 
25 testing, are generated. Thereafter, each candidate diagnosis is assigned a weight 
based in part on any of a number of factors not previously integrated into the 
weighting process, that weight indicating a level of confidence that the diagnosis is 
correct. For example, the weight of a diagnosis may be based on the combined 
unviolated utilization of the candidate diagnosis across all failing system tests. 

6 
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Alternately, or in addition, the weight may be based on the combined utilization of the 
candidate diagnosis among all passing system tests. Other factors that may be utilized 
instead of, or in addition to, those mentioned above, such as an observed variability of 
the results of passing and failing system tests, may form the basis of the weight of the 
5 candidate diagnosis as well. 

Generally speaking, the use of such factors in weighting a candidate diagnosis 
produces a more accurate indication of whether the associated diagnosis is correct. 
As a result, circumstances in which closely valued or identical weights for multiple 
candidate diagnoses result are significantly reduced. 
10 Other aspects and advantages of the invention will become apparent from the 

following detailed description, taken in conjunction with the accompanying drawings, 
illustrating by way of example the principles of the invention. 



1 5 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a high-level flow chart of a method of diagnosing dynamic system 
faults according to an embodiment of the invention. 

FIG. 2 is a flow chart of a method according to embodiments of the invention 
20 of generating and weighting candidate diagnoses possibly responsible for an observed 
test failure. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

25 

A high-level description of a method 100 according to an embodiment of the 
invention for diagnosing complex faults of a system under test (SUT) is shown in 
FIG. 1. In general terms, given a model of the SUT, a model for each passing and 
failing test performed on the SUT, and the results of those tests, at least one candidate 

7 
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diagnosis is generated (step 1 10). Typically, the candidate diagnosis is initially in the 
form of a hitting set, as described above. Using the same models and test results, a 
weight is then assigned to each diagnosis (step 120). Factors that may contribute to 
the weight include values that have not been previously considered in the prior art, 
5 such as the combined unviolated utilization of each diagnosis across all failing tests 
and the combined utilization, whether violated or unviolated, of the corresponding 
diagnosis across all passing tests. Other possible factors not considered in prior art 
diagnostic system are described below. Other factors typically taken into account, 
including the failure rate of each component of a candidate diagnosis, may also 

10 contribute to the weight of each diagnosis, as shown below. Once all factors are 
considered, the highest weight calculated indicates the most likely candidate 
diagnosis responsible for the observed failing tests. 

Typically, embodiments of the present invention are implemented in software 
executed on a general-purpose single computer system, a distributed general-purpose 

15 computing system, a special-purpose computing system designed specifically for 
dynamic system testing, a microprocessor-based computing system, or any other 
electronic system capable of executing an algorithm represented in software. 
Additionally, although software is the most likely medium, embodiments according to 
the invention also may be implemented in firmware for an embedded controller, or in 

20 hardware, such as in an application-specific integrated circuit (ASIC), for example. 
To further explain the salient features of various embodiments of the 
invention, a detailed description and some possible alterations of a specific 
embodiment are provided below, followed by some examples of how that 
embodiment applies to some simplified test cases. A pictorial representation of the 

25 general flow of a particular diagnosing method 200 is shown in FIG. 2. 

As noted above, generation of candidate diagnoses (step 110 of FIG. 1) is 
based on the system and test models, and the observed test results. In this particular 
example, the process begins with each test Ti executed on the SUT being identified 
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with a set of "coverage elements," with each coverage element being either a 
"structural element coverage" or a "shared function." 

A structural element coverage is an unnamed pair, consisting of (1) a 
component or subcomponent of a system under test (SUT), termed a "structural 
5 element," and (2) a utilization factor within the range of [0,1], which indicates the 
relative amount of the associated component or subcomponent that is utilized, or 
"covered," by the test T\. As an example, a structural element coverage that indicates 
a 70% utilization of a component Ci may be denoted (Ci, 0.7). Likewise, a structural 
element coverage involving a 30% utilization of a subcomponent sci of component C 2 
. 10 may be indicated as (C2::sci, 0.3). 

Since a structural element coverage is unnamed, a single structural element 
coverage may not appear in more than one test, and hence, cannot be violated (i.e., 
cannot be utilized in both a passing test and a failing test). For example, multiple 
instances of (C2::sci, 0.3) within multiple tests are not violated since each of the 
15 instances of that structural element coverage do not necessarily cover the same 
portions of the subcomponent C2i:sci. 

A shared function is a named set of structural element coverages associated 
with a particular operation. For example, a shared function sfl which involves the 
two above-mentioned shared element coverages may be denoted as sfl = ((Ci, 0.7), 
20 (C2i:sci, 0.3)). The same shared function may be present in one or more tests due to 
its being associated with a particular name, and thus, by definition, may be violated 
(i.e., may be employed in both a passing test and a failing test). Shared functions are 
essentially identical to the "operations" concept noted above. 

When a particular test Tj fails, the set of coverage elements associated with 
25 that test is termed a "coverage element conflict set" (CECS) (step 210 of FIG. 2). 

Similarly, the structural elements (i.e., component or subcomponent) that are utilized 
in the coverage element conflict set collectively comprise a "structural element 
conflict set" (SECS) (step 210). 

9 
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As an example, assume failing tests Ti and T 2 , and passing tests T 3 , T 4 and T 5 , 
as shown in the following simplified SUT and test models: 



Components: 
5 Ci: Subcomponents: None 

C2: Subcomponents: sc 2 , sc 3 , SC4 

Shared Functions: 

sfl = {(C,,0.6)} 
10 sf2= {(C 2 ::sc 2 , 0.4)} 

sf3 = {(C 2 ::sc 3 , 0.3)} 
sf4= {(C 2 ::sc 3 , 0.6)} 

Tests: 

15 Ti = {sfl, sf3}; Result = Fail 

T 2 = {sfl, sf2, sf4, (C 2 ::sc 4 , 0.7)}; Result = Fail 

T 3 = {sfl, sf4, (C 2 ::sc 2 , 0.4), (C 2 ::sc 3 , 0.5), (C 2 ::sc 4 , 0.2)}; 

Result = Pass 
T 4 = {sf3, (Ci, 0.9)}; Result = Pass 

20 T 5 = {sfl , sf2} ; Result = Pass 

In this case, the coverage element conflict sets for failing tests Ti and T 2 are 
CECS(T,) = {sfl, sfi} and CECS(T 2 ) = {sfl, sf2, sf4, (C 2 ::sc 4 , 0.7)}. 

Continuing with this example, a "structural element conflict set" (SECS) is 
25 also associated with each failing test. A structural element conflict set is a set of 

structural elements wherein each element has some coverage, by way of a structural 
element coverage or a shared function, in the coverage element conflict set for a 
particular failing test. In this case, the structural element conflict sets for failing tests 
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Ti and T 2 are SECS(Ti) = {d, C 2 ::sc 3 } and SECS(T 2 ) = {C b C 2 ::sc 2 , C 2 ::sc 3 , 
C 2 ::sc4}. 

Based on these conflict sets, associated "hitting sets" may then be. derived 
therefrom (step 220 of FIG. 2). A "coverage element hitting set" (CEHS), for 
5 example, is a set of coverage elements that has a non-empty intersection with every 
coverage element conflict set. In other words, a coverage element hitting set must 
have at least one coverage element from each coverage element conflict set. Thus, for 
the above example, one possible coverage element hitting set is CEHS = {sfl} since 
sfl is a coverage element of both failing tests Ti and T 2 . This set may also be referred 

10 to as a minimal coverage element hitting set, as it contains the least number of 
elements possible, which is one in this case. Another possible coverage element 
hitting set that is non-minimal is CEHS = {sfl, sf2}. 

Also, one or more "structural element hitting sets" may be identified from the 
structural element conflict sets identified for each failing test (step 220 of FIG. 2). 

15 Analogous to coverage element hitting sets, a structural element hitting set (SEHS) 
has a non-empty intersection with every structural element conflict set. Put another 
way, each structural element hitting set is required to have at least one structural 
element from each structural element conflict set. Using the same example shown in 
FIG. 1, two possible minimal structural element hitting sets are {Ci} and {C 2 ::sc3}. 

20 The concept of conflict sets and hitting sets has been the subject of significant 

research. These concepts, as well as potential algorithms for generating those sets, is 
discussed in greater detail, for example, by R. Reiter in "A theory of diagnosis from 
first principles," published in Artificial Intelligence 32(l):57-96 (1987). 

Closely related to each structural element hitting set is a "replaceable 

25 component hitting set," which is a structural element set that includes only 

components, as opposed to subcomponents. For example, the replaceable component 
hitting set (RCHS) associated with a structural element hitting set SEHS = {C 2 ::sc 3 } 
is RCS = {C 2 }. Replaceable component hitting sets are valuable from the standpoint 
that each element of such a set is an identifiable component which may be replaced if 
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found to be defective. Each candidate diagnosis to be weighted according to 
embodiments of the present invention, discussed immediately hereafter, is a 
replaceable component hitting set, and also essentially a structural element hitting set. 
Given the necessary hitting sets previously generated (step 220), embodiments 
5 of the present invention discussed herein generate scores for those hitting sets prior to 
generating weights for the replaceable component hitting sets, or candidate diagnoses. 
In other embodiments, scores may be directly generated for the replaceable hitting set 
using methods similar to those described below. While rather specific scoring 
formulae are explained below, these formulae are meant to be exemplary in nature; 

10 similar functions that are enveloped by the claims to follow may also be implemented 
within the scope of the present invention. 

First, each coverage element hitting set is scored relative to a pertinent 
structural element hitting set (i.e., the structural elements are covered or utilized by 
one or more coverage elements) (step 230). Thus, multiple scores may be associated 

15 with a single coverage element hitting set. The method of scoring is dependent to a 
degree on whether any of the coverage elements of a particular coverage element 
hitting set is "violated" (i.e., whether any of the coverage elements are utilized in both 
passing and failing tests). If any of the coverage elements of the coverage element 
hitting set are employed in both passing and failing tests, then the score of the 

20 coverage element hitting set, given a particular structural element hitting set, depends 
in part upon a combined, or "average," "expected" coverage (EC) of all structural 
elements in the given structural element hitting set for all coverage elements of the 
coverage element hitting set that, in turn, are utilized in failing tests and not in passing 
tests (i.e., the combined "unviolated" utilization of the structural element hitting set). 

25 In addition, that score is also proportional to the combined "observed shared function 
variability" (OSFV) of "pass-fail conflicted" shared functions (i.e., shared functions 
employed in both passing and failing tests) that are elements of the coverage element 
hitting set: 



AGILENT CASE #10031229-1 



12 



Score C EHs(CEHSi | SEHSj) = n OSFV 

VPass-fail conflicted SFeCEHSj 

* 0 + ( Z EC(SE)vCEeCEHSieFailingtestsePassing tests / | SEHSj | )) 
VSEe SEHSj 



The expected coverage of a single structural element that is utilized in a single 
coverage element is just the coverage noted within the associated structural element 
coverage, as mentioned above. If a structural element is utilized in two coverage 
elements, the expected coverage of the structural element is defined in the disclosed 
10 embodiments as the two coverages added together, minus an expected intersection, or 
assumed "overlap": 

EC(coveragei, coverage 2 ) = coverage i + coverage2 - (coverage i* coverage2> 

15 If three or more coverages are to be combined, the same method may be 

applied in an iterative fashion. For example, three coverages may be combined by 
first combining any two, and then combining the third thereafter: 



20 



EC(coveragei, coverage2, coverages) = EC((EC(coveragei, coverage2)), coverages) 



In alternate embodiments, other mathematical methods of "combining" two 
such coverage values other than averaging may be employed. For example, dividing 
the sum of the expected coverage of the structural elements by the number of 
elements of the structural element hitting set may not be necessary in some 
25 embodiments. Also, merely counting the number of structural elements involved, 

instead of adding the expected coverages of those structural elements, may be all that 
is required in some circumstances. Other variations not listed herein are also 
possible. 

Additionally, not all coverages of a particular coverage element are included 
30 in the expected coverage computation. Whether a coverage for a particular coverage 
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element is included in the overall expected coverage depends on whether the coverage 
element is a structural element coverage or a shared function. All coverages 
associated with structural element coverages do not necessarily coincide and, hence, 
are included in the calculation. However, since the coverage associated with a shared 
5 function is considered to be approximately the same each time the shared function is 
utilized, only one instance of a coverage for a shared function is included in the 
calculation, even if multiple uses of the shared function across multiple tests are 
involved. 

The observed shared function variability of a particular pass-fail conflicted 
10 shared function is essentially an operation penalty, similar to what has been provided 
for in previous diagnostic systems. However, instead of employing a constant "input 
variability" for this value, as has been done in the past, the observed shared function 
variability depends upon the input variability, adjusted by the number of passing and 
failing tests observed that employ the shared function, making the observed 
15 variability a value that more closely resembles the confidence that ordinarily would 
be placed in the observed test results: 

OSFVsf = min(0.9, InputVariabilitysF * 2, InputVariabilitysF 
* (1 + O.lflFailedTests | SF e failedtest) - 1) / (PassingTests | SF e passingtest|)) 

20 

In this embodiment, the observed shared function variability decreases greatly 
with the number of passing tests, and increases slowly by about ten times with the 
number of failed tests observed. In alternate embodiments, other methods of allowing 
the number of failed and passing tests to affect the observed shared function 
25 variability may also be employed. Also, instead of employing the number of failed 
and passing tests employing the shared function, other methods of quantifying the 
observed shared function variability may be used. For example, the utilization of the 
structural elements by the shared function may be used to quantify the observed 
shared function variability. 

14 
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In the specific calculation shown above, the observed shared function 
variability is limited at its maximum by 0.9 and by twice the input variability of the 
shared function. In this particular case, the use of such limits, referred to herein as 
mathematical "governors," restricts the impact of the observed shared function 
5 variability on the ultimate weighting of the candidate diagnosis. In general, 
governors limit the effect that uncertain or incomplete knowledge, such as why 
particular tests pass or fail utilizing the same shared function, has on the ultimate 
weighting. Governors, as will be indicated below, may also limit the minimum value 
of a weighting factor, the size of a factor relative to another, and so on. Furthermore, 

10 use of governors as shown is not mandatory in other embodiments of the present 
invention. Additionally, governors may not be limited to those instances indicated, 
but may be employed with any factor of the disclosed computation to limit that 
factor's effect on the overall weighting of the candidate diagnoses. 

As a result, the score of a coverage element hitting set, and ultimately the 

15 weight of an associated candidate diagnosis, is based in part on the combined, 

unviolated utilization of that diagnosis across one or more failing tests. As shown 
above, the observed variability of the shared functions involved also affects that 
score. 

In alternate embodiments, observed variability may not be related solely to the 
2 o presence of pass-fail conflicted shared functions. For example, a subset of system 
tests might be dependent on each other in varying ways, such as by covering similar 
physical portions of a particular component. If one system test of that subset fails 
while another passes, some level of variability has been observed, thus reducing the 
level of confidence in the test results. By quantifying the level of dependence 
25 between tests, the observed variability involved may also be quantified, allowing for 
some mathematical reduction in the weight of a candidate diagnosis associated with 
those tests. 

Once the coverage element hitting sets are scored, a score is computed for 
each of the structural element hitting sets, given the entire set of coverage element 
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hitting sets (step 240 of FIG. 2). The score of each structural element hitting set is 
inversely dependent upon the combined passing test utilization of the structural 
element hitting set. In other words, the higher the utilization of each structural 
element of the hitting set by a passing test, the lower its score indicating that the 
5 structural element is faulty. Also, the score of a structural element hitting set is 
proportional to the maximum score for any coverage element hitting set defined for 
that structural element hitting set: 

Score SE Hs(SEHSi | {CEHS}) 

10 = max(l - ( £ EC(SE) VC E e CEHS e Passing tests / I SEHSj |), 0.01) 

VSEeSEHSi 

* max(Score C EH S (VCEHSG {CEHS}|SEHSO) 

The expected coverage portion of the score for a structural element hitting set 
15 serves as a combined degree of utilization of the structural elements of the hitting set 
over all passing tests. The higher the coverage by the passing tests, the lower the 
overall score for the structural element hitting set. Also, this portion of the score is 
limited by a governor so that those passing tests cannot provide more than a 99% 
confidence that the structural element hitting set is not defective. Such a governor 
20 may also be eliminated in alternate embodiments of the current invention. 

Once calculated, the scores for the structural element hitting sets are combined 
with the failure rate for a "replaceable component hitting set" (RCHS), or candidate 
diagnosis, represented by the structural element hitting set to obtain a score for the 
replaceable component hitting set (step 250 of FIG. 2). Such scoring depends on 
25 whether any subcomponents are included in the structural element hitting set. 

If no subcomponents are identified in the particular structural element hitting 
set, then the structural element hitting set and the replaceable component hitting set 
are one and the same. In that case, the failure rate of the replaceable component 
hitting set is multiplied by the score of that hitting set: 

30 
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Score R cHs(RCHSi | |subconiponents| = 0) 

= FailRate(RCHSi) * Score SE Hs(RCHSi | {CEHS}) 



As mentioned above, the failure rate of a replaceable component hitting set 
5 may include the manufacturing failure rate of the component itself, the failure rate of 
that component due to process errors, and the like. 

Normally, a replaceable component hitting set consists of a single component, 
making the failure rate of the replaceable component hitting set equivalent to the 
failure rate of that component. In cases where more than one component is involved, 
10 the failure rates of the separate components may be simply multiplied together if 
failures of the components are highly independent. 

In the case that subcomponents are named in a structural element hitting set, 
the replaceable component hitting set has as its elements the components which 
include the subcomponents of the structural element hitting set. The score for this 
15 type of replaceable component hitting set in the present embodiment is proportional to 
the maximum score for structural element hitting sets that include subsets of the 
subcomponents of the replaceable component hitting set. In addition, that score is 
also dependent upon the average passing confidence of subcomponents not existing in 
any structural element hitting set. In other words, subcomponents that reside both 
20 within and outside the structural element hitting set are considered: 

Score RC Hs(RCHSi | (subcomponents eRCHS* | > 0) 
= FailRate(RCHSj) 

* max(Score SE Hs(VSEHS having subcomponents eRCHS* | {CEHS})) 
25 * max(l - X EC(subcomponent) / [subcomponents eRCHSj£ any SEHS|, 0.5) 

VsubcomponentseRCHSieany SEHS 

Additionally, a governor (i.e., 0.5 in the above formula) is used in this 
implementation which limits the effect of the subcomponents of the replaceable 
3 0 component hitting set that reside outside of the associated structural element hitting 
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set in reducing the weight of the candidate diagnosis. Use of that particular governor, 
or any governor at all, is not required. 

According to the above equation, average passing utilization of 
5 subcomponents residing outside any hitting set is considered. In alternate 
J embodiments, utilization of those subcomponents by failing tests, or some 

combination of both failing and passing utilization, may also be employed to adjust 
the score of the replaceable component hitting set. 

Also in this particular embodiment, only subcomponents outside any structural 
10 element hitting set are considered in reducing the score of the replaceable component 
hitting set. In alternate embodiments, subcomponents of the RCHS that are outside 
the structural element hitting set associated with the RCHS, but within another SEHS, 
may be considered. 

Given the above formulae utilized in the particular embodiment of the present 
15 invention, describing a few simple test cases aids in explaining how the above 

equations are applied in particular situations, and how the various equations interact 
to obtain a weight for each candidate diagnosis. 

Example 1 

20 In Example 1, described below, weights for a set of candidate diagnoses 

involving only failing tests are calculated. Models for the components of the SUT, 
the shared functions employed, and the tests executed are as follows: 

^Components: 

25 Ci: FailRate = 0.01; Subcomponents: sci 

C2: FailRate = 0.01; Subcomponents: None 

Shared Functions: 

sfl = {(Ci::sci, 0.5)}; Input variability = 0.1 

18 

AGILENT CASE #10031229-1 



s£2 = {(Ci::sci, 0.9)}; Input variability = 0.1 
sf3 = {(C 2 , 0.1)}; Input variability = 0.1 



Tests: 

5 Ti = {sfl , sf3} ; Result = Fail 

T 2 = {sf2, sf3, (Ci::sci, 0.2)}; Result = Fail 
T 3 = {sf3, (Ci::sci, 0.1)}; Result = Fail 

As no passing tests are involved, no shared function violations are possible. 
10 As a result, observed shared function variability (OSFV) is not considered, so that 
factor has the value of one. Possible coverage element hitting sets and structural 
element hitting sets are generated as described earlier: 

CEHS = {{sG}, {sfl, (C,::sc u 0.2), (d::sci, 0.1)}, {sfl, sf2, (Ci::se b 0.1)}, 
is {sfl, s£2, (Ci::sci, 0.1), (Ci::sci, 0.2)}} 
SEHS={{C 2 }, {Ci::sci}} 

Since sG is the only shared function that utilizes component C 2 , which is a 
structural element hitting set, only the coverage from the coverage element hitting set 
20 CEHSj = {sf3} figures into the scoring for that structural element hitting set. The 
three remaining coverage element hitting sets, as they only cover the subcomponent 
Ci::sci, contribute to the score for that structural element hitting set. 

Given SEHSi = {C 2 }, the score for CEHSi = {sG} is calculated as follows: 

25 Score CE Hs(CEHSi | SEHSi) = 1 + EC sf3 (C 2 ) = 1.1 

In turn, since there are no passing tests involved, the score for SEHS {C2} is 
the maximum score for an associated coverage element hitting set, which in this case 
is simply the expected coverage of sG over C 2 : 
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Score SE Hs(SEHSi | {CEHS}) = Score C EHs(CEHS } | SEHSi) =1.1 



Similarly, given SEHS 2 = {Ci::sci}, the three remaining CEHS, which are 
5 associated with SEHS 2 , (CEHS 2 = {sfl, (Ci::sci, 0.2), (C t ::sci, 0.1)}, CEHS 3 = {sfl, 
sf2, (C,::sci, 0.1)}, and CEHS 4 = {sfl, sf2, (C,::sc,, 0.1), (C,::sc,, 0.2)}) are 
calculated in a similar fashion, with the separate coverages of that single structural 
element combined as described above: 

10 Score C EHs(CEHS 2 | SEHS 2 ) = 

1 + ECvce€Cehs2 (Ci::sci) = 1 + EC(0.5, 0.2, 0.1) = 1.640 
Score CE Hs(CEHS 3 | SEHS 2 ) = 

1 + EC VC E€CEHS3 (Ci::sci) = 1 + EC(0.5, 0.9, 0.1) = 1.955 
Score CE Hs(CEHS 4 | SEHS 2 ) = 
15 1 + EC V c Ee cEHS4(C,::sci) = 1 + EC(0.5, 0.9, 0.1, 0.2) = 1.964 

Again, since there are no passing tests involved, the score for SEHS {Ci::sci} 
is the maximum score of the three associated coverage element hitting sets: 

20 Score SE Hs(SEHS 2 | {CEHS}) 

= max(Score CE Hs(CEHS 2 | SEHS 2 ), Score CE Hs(CEHS 3 | SEHS 2 ), 
Score C EHs(CEHS 3 1 SEHS 2 )) = 1.964 

Finally, the scores for the two replaceable component hitting sets, {Ci} (the 
25 replaceable component for SEHS 2 = {Ci::sc ( }) and {C 2 }, are calculated. For 
component C 2 , no subcomponents are involved, so the score is the previously 
calculated score for SEHSi times the component failure rate: 

Score RCH s({C 2 }) = FailRate(C 2 ) * Score SE Hs(SEHSi | CEHSi) = 0.01 * 1.1 = 0.01 1 
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For the replaceable component hitting set {Ci}, a subcomponent Ci::sci is 
involved. However, since no subcomponents of Ci reside outside of SEHS2 or 
SEHS3, the score for {Ci} is reduced to the no-subcomponent case: 



Score RCH s({Ci}) = FailRate(d) * Score S EHs(SEHS 2 1 {CEHS}) = 0.01 * 1.964 
= 0.01964 

Thus, given the scores for the two replaceable component hitting sets, 
10 component Ci is calculated to be the most likely cause of the failing tests Ti, T2, and 
T 3 . This result is due to a great extent to the low utilization by sf3 of Ci compared to 
the utilization of €2 by shared functions sfl and sf2. 

Example 2 

15 In Example 2, a mixture of passing and failing tests is used to indicate how the 

scoring functions take such results into account. The models for the tests, 
components and shared functions are listed: 

t Components: 

20 Q: FailRate = 0.01; Subcomponents: sci, SC2, SC3 

Shared Functions: 

sfl = {(Ci::sci, 0.4)}; Input variability = 0.1 
sf2 = {(Ci::sc 2 , 0.2)}; Input variability = 0.1 
25 sf3 = {(Ci::sc 2 , 0.3)}; Input variability = 0.1 

sf4 = {(Q::sci, 0.4)}; Input variability = 0.1 
sf5 = {(Ci::sc 3 , 0.7)}; Input variability = 0.1 



Tests: 
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Ti = {sfl, sf4, sf5, (Ci::sc 3 , 0.1)}; Result = Fail 
T 2 = {sfl, sf2, sf3, (Ci::sc 2 , 0.1)}; Result = Pass 

First, possible coverage element hitting sets and structural element hitting sets 
5 for this example are generated: 

CEHS = {{sfl}, {sf4}, {sf5}, {(C,::sc 3 ,0.1)}} 
SEHS = {{Ci::sci}, {d::sc 3 }} 

io To simplify the example for the purpose of facilitating understanding of the 

present invention, the number of coverage element hitting sets is reduced below to 
two: CEHSi = {sf4} and CEHS 2 = {sf5}. 

As before, scoring of each of the coverage element hitting sets, CEHSi and 
CEHS2, neither of which involve pass-fail conflicted shared functions, is performed 

15 ^relative to the structural element hitting sets with elements that are covered by each 
CEHS. In this case, sf4 utilizes only Ci::sci (making up SEHSi), and sf5 covers only 
Ci::sc3 (SEHS2). Hence, only those combinations need be considered when 
calculating CEHS scores: 



20 ScorecEHsCCEHSj | SEHS,) = Jl OSFV * (1 + (EC sf4 (Ci::sci))) 

VPass-fail conflicted SFeCEHS 1 

= 1 *(1 +0.4) = 1.4 



25 



Score CE Hs(CEHS 2 | SEHS 2 ) = n OSFV 

VPass-fail conflicted SFeCEHS2 



* (1 + EC(Ci::SC3)vCEeCEHSeFailingtests*Passing tests) 

= 1 * (1 + EC(EC sf5 (C^:sc 3 ), EC ( ci::sc3,o.i)(C,::sc 3 ))) = 1 + EC(0.7, 0.1) 
30 = 1 +0.73 = 1.73 

In scoring CEHS 2 , the coverages of both shared function sf5 and the structural 
element coverage (Ci::sc3, 0.1) within failing test Tj are combined. 
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The two structural element hitting sets are then calculated, using the CEHS 
scores just generated. For each SEHS, coverage of that hitting set by any passing 
tests reduces the score of that SEHS accordingly: 



10 
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Score SE Hs(SEHSi | {CEHS}) 

= max((l - EC(Ci::sci)vce 6 cehs € t2), 0.01) 

* max(Score CE Hs( VCEHS e {CEHS } | SEHS 0) 

= max((l -EC s n(Ci::sci)), 0.01) * Score CE Hs(CEHSi | SEHSi) 

= max(l - 0.4, 0.01) * 1.4 = 0.6 * 1.4 = 0.84 

In scoring SEHSi, the coverage of the structural element Ci::sci by sole 
passing test T 2 is by sfl, providing a coverage of 0.4. That coverage acts to decrease 
the score for SEHSi due to lowered confidence in that hitting set being the cause of 
the failing test Ti. 



Score SE Hs(SEHS 2 | {CEHS}) 

= max((l - EC(Ci::sc 3 )vcE e cEHSGT2), 0.01) 
* max(Score CE Hs(VCEHSe{CEHS} | SEHS 2 )) 
= max(l - 0, 0.01) * Score C EHs(CEHS 2 | SEHS 2 ) 
20 = max(l,0.01)* 1.73= 1 * 1.73 = 1.73 

Contrary to SEHSi, SEHS 2 contains structural element Ci::sc 3 , which is not 
covered by passing test T 2 , so the score for SEHS 2 is not reduced as a result. 

As a result of these structural element hitting set scores, the score for the 
25 associated replaceable component hitting set {Ci} is computed, keeping in mind that 
subcomponents of Ci make up both SEHSi and SEHS 2 : 

Score RCH s({Ci}) 

= FailRate(d) 
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* max(Score SE Hs(SEHSi | {CEHS}), Score S EHs(SEHS 2 1 {CEHS})) 

* max(l -ECCECsnCC^isci), EC sf 3(Ci::sc 2 ), EQci^o^CQ:^)), 0.5) 
= 0.01 * max(1.4, 1.73) * max(l - EC(0.2, 0.3, 0.1), 0.5) 

= 0.01 * 1.73 * max (1 -0.496, 0.5) = 0.01 * 1.73 * 0.504 = 0.0087192 

5 

Although this particular example results in a single replaceable component 
hitting set, making the actual selection of the defective component trivial, the above 
computation for the score for RCHS = {C\ } shows the reducing effect that the 
subcomponent Ci::sc2 that is not a part of any structural element hitting set. The 
10 coverage of that component by shared functions sfl and s£2, as well as structural 
element coverage (Ci ::sc 2 , 0. 1) within passing test T 2 reduces the weight otherwise 
given for d if that subcomponent had not been taken into account. 

Example 3 

is The next example, although it also results in a single possible replaceable 

component hitting set, exemplifies the presence of multiple subcomponent structural 
element hitting sets in conjunction with passing coverage of subcomponents outside 
of any SEHS. The models for this particular model are as follows: 

20 Components: 

Ci: FailRate = 0.01; Subcomponents: sc 3 , sc 4 , sc 5 , sc 6 

Shared Functions: 

sf4 = {(Ci ::sc 4 , 0.6)} ; Input variability = 0. 1 
25 sf5 = {(Ci::sc 5 , 0.7)}; Input variability = 0.1 

sf6 = {(Ci::sc 6 , 0.8)}; Input variability = 0.1 

Tests: 

Ti = {sf5}; Result = Fail 
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T 2 = {sf6}; Result = Fail 

T 3 = {sf4, sf5, sf6, (Ci::sc 3 , 0.1)}; Result = Pass 



Given this model, a single coverage element hitting set CEHSi = {sf5, sf6} is 
5 possible since both of these shared functions are necessary to cause tests Ti and T 2 to 
fail, and no other coverage elements are required for those failures. Furthermore, the 
only coverage elements utilized within CEHSi are Ci::sc 5 (by sf5) and Ci::sc6 (by 
sf6), resulting in a single structural element hitting set SEHSi = {Ci::sc5, Ci::sc6}. 

In computing the score for CEHSi, the fact that sf5 and sf6 are both pass-fail 
10 conflicted raises the need for computing the observed shared function variability 

(OSFV) of those shared functions. Also, since there are no other coverage elements 
within CEHSi, there is also no coverage of those elements by failing tests that are not 
also covered by passing tests: 

is Score CE Hs(CEHSi | SEHSi) = n OSFV 

VPass-fail conflicted SF e CEHS 1 

* 0 + ( Z EC(SE)vCEeCEHSleFailingtests«£Passing tests) / | SEHS] | ) 
VSEeSEHSl 

= OSFV(sf5) * OSFV(sf6) * (1 + 0) { 
20 = min(0.9, InputVariability(sf5) * 2, InputVariability(sf5) * (1 + 0.1 * (1 - 1)/1) 
* min(0.9, InputVariability(sf6) * 2, InputVariability(sf6) * (1+ 0.1 * (1 - 1)/1) 
= min(0.9, 0.1 * 2, 0.1) * min(0.9, 0.1 * 2, 0.1) = 0.1 * 0.1 =0.01 

Next, the score for the structural element hitting set SEHSi is computed, 
25 taking into account the fact that the structural elements Ci::sc 5 and Ci::sc6 are covered 
by passing test T3, thus reducing the score for SEHSi: 

Score SE Hs(SEHSi | {CEHS}) 

= max(l - ((EC T 3(Ci::sc 5 ) + EC-n(Ci::sc 6 )) / 2), 0.01) 
30 * max(Score C EHs(CEHSi | SEHSi)) 
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= max(l 
= max(l 



-((0.7 + 0.8)/ 2), 0.01) 0.01 
-0.75, 0.01)* 0.01 =0.0025 



Given the SEHSi score, the score for the sole replaceable component hitting 
5 set {Ci} is calculated, taking into account the fact that the two subcomponents of Ci 
residing outside SEHSi, Ci::sc 5 and Cj::sc6, were involved in passing tests: 

Score RCH s({Ci}) = FailRate({C,}) 

* max(Score S EHs(SEHS,)) 
io * max(l - ((EC S f 4 (Ci::sc 4 ) .+ EC ( ci::sc3,o.i)(Ci::sc 3 )) / 2), 0.5) 

= 0.01 * 0.0025 * max(l - ((0.6 + 0.1) / 2), 0.5) = 0.00001625 

Therefore, the reductions in scoring due to the pass-fail conflicts of the shared 
functions sf5 and sf6, as well as the coverage of subcomponents Ci::sc3 and d::sc4 in 
15 passing test T3, contribute to a relatively low score for the replaceable component Ci. 

Example 4 

In the fourth and final example, the effect of shared function penalties is 
further explored, as well as the effect of shared functions having coverage over 
20 multiple components. The specific details of the relevant models for this example are 
listed below: 

Components: 

Ci: FailRate = 0.01; Subcomponents: None 
25 C2: FailRate = 0.01; Subcomponents: None 

Shared Functions: 

sfl = {(Ci, 0.5), (C 2 , 0.9)}; Input variability = 0.1 
sf2 = {(Ci, 0.6), (C 2 , 0.1)}; Input variability = 0.1 
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Tests: 

5 T 3 = 



{sfl,sf2}; Result = Fail 
{sfl}; Result = Fail 
{sfl}; Result = Pass 



Given these models, the only relevant coverage element hitting sets are 
CEHSi = {sfl} and CEHS2 = {sfl, sf2}, as they are the only coverage elements 
utilized in the failing tests Ti and T 2 . Also, since components Ci and C2 are both used 

10 in each of the shared functions, each component makes up a separate hitting set. As a 
result, SEHSi = {Ci} and SEHS 2 = {C 2 }. A structural element hitting set combining 
the two is possible, but may be eliminated since the combination of both components 
failing is less likely than one or the other being exclusively defective. 

As shown above, the scores for each coverage element hitting set is generated. 

15 Since CEHSi and CEHS 2 each cover both SEHSi and SEHS 2 , a total of four CEHS 
scores are determined: 

Score CE Hs(CEHS, | SEHS,) = OSFV(sfl) * (1+ EC(Ci)vc Ee CEHSieTi ,12*13) 
= min(0.9, Input Variability(sfl) * 2, Input Variability(sfl) * (1 + 0.1 * (2 - 1)/1) 
20 *(l+0) 

= min(0.9, 0.1 * 2, 0.1 * (1.1)) = 0.11 

Score CE Hs(CEHS 2 | SEHS,) = OSFV(sfl) * (1 + EC(C,)v C E e cEHS2 e Ti > T2eT3) 

= min(0.9, InputVariability(sfl) * 2, InputVariability(sfl) * (1 + 0.1 * (2 - 1)/1) 

25 * (1 + ECscCC)) 

= min(0.9,0.1 *2,0.1 * (1.1)) * (1 +0.6) = 0.11 * 1.6 = 0.176 

ScorecEHs(CEHSi | SEHS 2 ) = OSFV(sfl) * (1 + EC(C 2 )vc Ee cEHSi S T!,T2« £ T3) 

= min(0.9, InputVariability(sfl) * 2, InputVariability(sfl) * (1 + 0.1 * (2 - 1)/1) 
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*(1 +0) 

= min(0.9, 0.1 * 2, 0.1 * (1.1)) = 0.1 1 



Score C EHs(CEHS 2 | SEHS 2 ) = OSFV(sfl) * (1 + EC(C 2 ) V cEeCEHS2 e -n,T2 e T 3 ) 
5 = min(0.9, InputVariability(sfl) * 2, InputVariability(sfl) * (1 + 0. 1 * (2 - 1)/1) 
*(l+EC sf2 (C 2 )) 

= min(0.9, 0.1 * 2, 0.1 * (1.1)) * (1 + 0.1) = 0.1 1 * 1.1=0.121 

Again, with the calculated CEHS scores, the scores for SEHSi and SEHS 2 are 
10 then computed: 

Score SEH s(SEHSi | {CEHS}) 

= max(l - EC(Cl)vCEeCEHSePassing tests), 0.01) 

* max(Score C EHs(VCEHSe{CEHS} | SEHSi)) 
15 = max(l - ECsfi(C,), 0.01) * Score CE Hs(CEHS 2 | SEHSi) 

= max (1 - 0.5, 0.01) * 0.176 = 0.5 * 0.176 = 0.088 

Score SE Hs(SEHS 2 | {CEHS}) 

. = max(l - EC(C 2 )vCEeCEHSePassing tests), 0.01) 

20 * max(Score CE Hs(VCEHSG {CEHS} | SEHS 2 )) 

= max(l - EC s n(C 2 ), 0.01) * Score CE Hs(CEHS 2 | SEHS 2 ) 
= max (1-0.9, 0.01)* 0.121 =0.1 * 0.121 = 0.0121 



Since no subcomponents are involved in this example, the resulting score for 
25 each of the two replaceable component hitting sets {Ci } and {C 2 } is the score of the 
corresponding structural element hitting set, multiplied by the failure rate of the 
component: 

Score RC Hs({C 1 }) = FailRate({C 1 }) * Score SE Hs(SEHS 1 | {CEHS}) 

28 

AGILENT CASE #10031229-1 



= 0.01 * 0.088 = 0.00088 

Score RC Hs({C 2 }) = FailRate({C 2 }) * Score S EHs(SEHS 2 | {CEHS}) 
= 0.01 * 0.0121 =0.000121 

5 

Thus, the score for replaceable component {Ci} is significantly higher that 
that for {C 2 }, which ultimately results from the penalty incurred in the score for 
SEHS 2 due in part to the heavy coverage of C 2 by pass-fail conflicted shared function 
sfl in passing test T3. 

10 Oftentimes, some parameters of the test models, such as shared function input 

variability, test utilization of a particular component, or a component's failure rate, 
may not be known with precision. In other words, only a vague sense of the range of 
a particular value, such as "low," "medium," or "high," may be articulated. In such 
cases, it may be desirable to allow a description of those values in such terms, and 

15 then assign those descriptions a particular numeric value in the actual scoring 

functions. For example, a "low" input variability may be associated with a value of 
0.1, with "medium" and "high" being assigned values of 0.5 and 0.9, respectively. 

In addition to the scoring factors discussed above, replacement cost may also 
be a factor in the scoring of a candidate diagnosis. Replacement cost is the cost to 

2 0 replace a component, and may include the cost of the replacement component plus the 
cost to effect the replacement. For example, a factor of one plus a "normalized cost" 
in the range of (0,1] would serve to introduce the cost of each candidate diagnosis, 
which may be the sum of the replacement cost of the replaceable components in that 
diagnosis, as a secondary scoring factor. Other ways of introducing replacement cost 

25 into the scoring functions described above may also be implemented in alternate 
embodiments of the present invention. While replacement cost itself does not 
necessarily affect the likelihood that a component is defective, using cost as a factor 
may help distinguish between two otherwise similarly-weighted candidate diagnoses 
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to help determine which of the two should be replaced first in attempting to eliminate 
the defect at the least cost. 

From the foregoing, embodiments of the invention provide an improved 
method for weighting candidate diagnoses involving complex system faults, and thus, 
5 an overall improved fault diagnosis method. Very detailed descriptions of specific 
test examples of the various embodiments have been supplied and explained. 
However, embodiments of the invention other than those shown above are also 
possible, some of which have been discussed herein. As a result, the invention is not 
to be limited to the specific forms so described and illustrated; the invention is limited 
10 only by the claims. 
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